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Box No. I 



Basis of this opinion 



1 . With regard to the language, this opinion has been established on the basis of the international application in the language in 
which it was filed, unless otherwise indicated under this item. 

I I This opinion has been established on die basis of a translation from the original language into the following language, 

, which is the language of a translation furnished for the purposes of international search (under Rules 12.3 
and 23.1(b)). 

2. With regard to any nucleotide and/or amino acid sequence disclosed in ttie international application and necessary to the 
■ claimed invention, this opinion has been established on the basis of: 

a. type of material 

I I a sequence listing 

1^ I table(s) related to flie sequence listing 



3. 



b. format of material 

I I in writtffli format 

I I in computer readable form 

c. time of filing/furnishing 

I I contained in the international application as filed. 

I I filed together with the international application in conq>uter readable form. 
j I furnished subsequently to this Authorify for the pwxwses of search. 

I [ In addition, in the case that more than one version or copy of a sequence listing and/or table relating tiioeto has been 
filed or furnished, the required statements that the information in the subsequent or additional copies is identical to 
that in the ^plication as filed or does not go b^ond the application as filed, as appropriate, were furnished. 
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applicability; citations and explanations supporting such statement 



1. Statement 



Novelty (N) 



Inventive step (IS) 



Industrial applicability (lA) 



Claims 
Claims 

Claims 
Qaims 

Claims 
Claims 



1-24 



1-24 



1-24 



YES 
NO 

YES 
NO 

YES 
NO 



2. Citations and explanations: 

The claimed invention relates to a system and a method that 
enable a user to interact with a virtual control panel. The 
invention can be applied to different industrial processes, 
where a user needs a control panel for interacting with a 
device or a system. The object of the invention is to 
provide a portable control panel that the user can carry 
with him/her- The interaction between the user and the 
devices is also simplified, since the interfaces can be 
standardized. The user only has to interact with a single 
interface, which changes its representation in dependence of 
the closest device. 



Docviments cited in the International Search Report 



Dl: EP 1271293 A2 
D2: WO 0192944 Al 



D3: US 20020044104 Al 
D4: US 20020191004 Al 



Documents D3 and D4 define the general state of the art 



Document Dl discloses a system and a method that enable 
user to interact with a virtual control panel. The 
wears a head-mounted see-through display that shows the user 
a view comprising the *^real world" and the virtual control 
panel (column 4, lines 24-30). The user can interact with a 
number of different devices, e.g. domestic appliances and 
vending machines (column 1, lines 6-18) . For example, the 
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Supplemental Box 



In case the space in any of the preceding boxes is not sufiicient. 
Continuation o£ Box V 

volume of a television set could be adjusted or the 
television set could be turned on or off (figs. 3a-3d) . The 
user could also receive valuable information about a certain 
device (column 4, lines 31-39) . Information about available 
coraniands associated with a certain device, i.e. the 
graphical interface that represents the control panel of the 
device, is stored in a database in the system (column 6, 
lines 12-25) . The user selects devices and commands by 
moving his/her view. However, the description also discloses 
that commands can be selected by pointing at virtual objects 
that are visible in the field of view of the user (column 1, 
line 32-coltamn 2, line 10) . 

Document D2 discloses a system for generating a virtual 
display- A mobile display device (6 in fig. 2a) is connected 
to a processing unit (5) , which is provided for generating a 
virtual image (abstract and claim 4) . The position of the 
virtual image is fixed and depends on a reference point that 
is specified by the user. 

The system according to document Dl is considered to 
represent the closest prior art. The difference between the 
invention according to claims 1 and 12 and Dl is that the 
system enables a user to interact with the virtual control 
panel using a ^pointing object". Further, the virtual 
control panel is projected in a fixed relation to an 
identification element. As mentioned earlier, the 
description in Dl also discloses that commands can be 
selected by pointing at virtual objects that are visible in 
the field of view of the user. Therefore, this difference is 
only considered to constitute an obvious, alternative 
embodiment for a person skilled in the art . 

The other difference between claims 1 and 12 and Dl, i.e. 
that the virtual control panel is projected in a fixed 
relation to an identification element, relates to the 
problem that the control panel could be in the field of 
vision of the user and thereby distract him /her. A person 
skilled in the art, facing this problem, finds a solution in 
document D2 . D2 teaches that a projected image can be 




Fomi PCT/ISA/237 (Supplemental Box) (January 2004) 



WRITTEN OPINION OF THE 
INTERNATIONAL SEARCHING AUTHORirY 



International application No. 
PCT/ IB2 0 0 4 / 004246 



Supplemental Box 



In case the space in any of the preceding boxes is not sufficient. 
Continuation of: BOX V 

fixed in relation to a reference point that the user 
specifies . 

Thus, a person skilled in the art, having the method and 
system in Dl as a starting point and aiming to solve the 
identified problem, would with the knowledge of D2 project 
the image in a fixed relation to a reference point. Thus, 
the skilled person arrives at the invention according to 
claims 1 and 12 • 

Since Dl and D2 both relate to the same technical field and 
no unexpected effect is obtained, the combination of what is 
known from Dl and D2 is considered obvious for a person 
skilled in the art. 

The invention according to claims 1 and 12 is thus not 
considered to involve an inventive step. 

Since the method lacks an inventive step, a computer 
program, executing said method, also lacks an inventive 
step. 



Dl discloses that the data from the devices could 
displayed in the interface and the ^pointing object" could 
be a part of the user's body. The user can interact with 
different devices and the devices are probably represented 
by different graphical interfaces. The devices could be 
provided with bar codes or visual tags to facilitate 
recognition (column 5, lines 9-27) . The user could, in one 
embodiment, manually choose which device to interact with 
(column 5, lines 39-48) . Further, the display unit in Dl is 
portable . 



the invention according to claims 3-8, 11, 12, 
14-16 and 18-20 lacks an inventive step. 

The subject-matter of the remaining claims, e.g. that the 
appearance of the virtual control panel could be modified 
and that the identification element is adapted to be carried 
by the user, is only considered to constitute details that 
are obvious for a person skilled in the art. 



/. 
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What is claimed in claims 2, 9, 10, 13, 17, 21 
thus not considered to involve an inventive step. 



and 22 is 



The invention is considered to be industrially applicable 
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1 . Statement 



Novelty (N) 



Claims 
Claims 



1-24 



YES 
NO 



Invoitive step (IS) 



Claims 
Claims 



1-24 
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NO 



Industrial applicability (lA) 



Claims 
Claims 
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2. Citations and explanations (Rnle 70.7) 

The Claimed invention relates to a system and a method that 
enable a user to interact with a virtual control panel. The 
invention can be applied to different industrial processes, 
where a user needs a control panel for interacting with a 
device or a system. The object of the invention is to provide 
a portable control panel that the user can carry with him/her. 
The interaction between the user and the devices is also 
simplified, since the interfaces can be standardized. The user 
only has to interact with a single interface, which changes 
its representation in dependence of the closest device. 

Docvmients cited in the International Search Report: 



Dl: EP 1271293 A2 
D2 : WO 0192944 Al 



D3: US 20020044104 Al 
D4: US 20020191004 Al 



Doctiments Dl and D2 have been reconsidered to define 
general state of the art. Docximents D3 and D4 also define 
general state of the curt. 



the 
the 



Document Dl discloses a system and a method that enable a user 
to interact with a virtual control panel. The user wears a 
head-mounted see-through display that shows the user a view 
comprising the «real world" and the virtual control panel 
(column 4, lines 24-30). The user can interact with a number 
of different devices, e.g. domestic appliances and vending 
machines (column 1, lines 6-18) . For example, the volume of a 
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television set could be adjusted or the television set could 
be turned on or off (figs, 3a-3d) . The user could also receive 
valuable information about a certain device (column 4, lines 
31-39) . Information about available commands associated with a 
certain device, i.e. the graphical interface that represents 
the control panel of the device, is stored in a database in 
the system (column 6, lines 12-25) . The user selects devices 
and commands by moving his/her view. However, the description 
also discloses that commands can be selected by pointing at 
virtual objects that are visible in the field of view of the 
user (column 1, line 32-column 2^ line 10) . 

Document D2 discloses a system for generating a virtual 
display. A mobile display device (6 in fig. 2a) is connected to 
a processing unit (5) , which is provided for generating a 
virtual image (abstract and claim 4) • The position of the 
virtual image is fixed and depends on a reference point that 
is specified by the user. 

The system according to document Dl is considered to represent 
the closest prior art. The invention according to claims 1 and 
12 differs from Dl in that the system enables a user to 
interact with the virtual control panel using a "pointing 
object". Further, the virtual control panel is projected in a 
fixed relation to an identification element, i.e. the control 
panel will follow the identification object as if it is 
physically attached to the element. 



The subject-matter of claims 1 
(Article 33 (2) PCT) . 



and 12 is therefore novel 



The difference between the claimed invention and Dl relates to 
the problem that the control panel could be in the field of 
vision of the user and thereby distract the user. The solution 
to this problem is proposed in claims 1 and 12 of the present 
application. The control panel will follow the identification 
object as if it is physically attached to the element. Hence, 
the user can put the virtual control panel away when doing 
other tasks. 

The cited prior art does not give any indication that would 



/ 
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lead a person skilled in the art to the claimed system and 
method. Therefore, the claimed invention is not obvious to a 
person skilled in the art. Consequently, the invention 
according to claims 1 and 12 is considered as involving an 
inventive step (Article 33(3) PCX). 

The remaining claims are dependent on claims 1 and 12, and as 
such also meet the requirements of the PCT with respect to 
novelty and inventive step. 
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5 A VIRTUAL CONTROL PANEL 

FIELD OF THE INVENTION 

The present Invention relates to a system and a method that en- 
10 ables a user to interact with a virtual control panel using a user 
controlled pointing object. The system comprises a first tracking 
unit adapted to capture data representing the position of the 
pointing object. 

15 The virtual control panel represents Interfaces for real-time hu- 
man-system interaction. The invention can be applied for differ- 
ent industrial processes where an operator needs a control pa- 
nel for interacting with a device or system regarding real-time 
data, historical data, simulations, test data, etc. The invention is 

20 particularly useful in automation applications. 

PRIOR ART 

Field operators (e.g. in process plants, manufacturing, at off- 
25 shore platforms and substations) need both to interact with sys- 
tems or devices and to have access to information and docu- 
mentations. Various user interfaces, both physical and software 
interfaces, towards devices and system are often designed dif- 
ferently and the field operators have to learn to use every single 
30 user interface system. In addition to the two-way interaction with 
devices/systems, the field operators need access to documenta- 
tion such as manual, historical data, maintenance & repair re- 
ports, P&IDs, etc. Even though the trend is that such documen- 
tation becomes electronically available, it often resides at differ- 
35 ent systems and locations. Therefore, the field operators have to 
plan their work tasks in advance, or they even have to interrupt 
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ongoing tasks in order to control the process/devices and find 
supportive information and documentation. 

Within manufacturing, a production line includes several robots 
5 which are controlled by separate controllers and teach pen- 
dants. The operators need to interact with the robots, for exam- 
ple, in order to check status, inspect the robots and make new 
programs. A general drawback is that the operators must 
change interaction device for each new robot even though sev- 
10 era I robots may perform a task together. 

Augmented Reality (AR) is a method of overlaying real world 
representations with computer-generated graphics. Ideally, for 
vision-based augmented reality, the user will not be able to rec- 

15 ognize the difference between the real and the computer- 
generated graphics, and thereby the user will get an improved 
perception of the real world environment. Today, augmented re- 
ality techniques are used in a number of applications. Examples 
of use are within media (weather reporting), medicine (visualisa- 

20 tion of internal organs), for collaborative environments (virtual 
meeting room) and in process Industries (for maintenance, ser- 
vice). 

A handheld/wearable control panel with interaction possibilities 
25 already exists and the interest of such devices Increases. The 
use of mobile phones and PDAs as the Interface towards sys- 
tems/devices is known. Also, tablet PCs, which are used, e.g. 
within hospitals, provide an interface that the user easily carries 
around and interacts with by touching the screen. The newest 
30 field that is related is virtual keyboards. A virtual keyboard is 
projected, e.g. at a table, and will typically be the input mean for 
a PDA, The user touches/presses the keys and the system rec- 
ognizes the specific touched keys like a standard keyboard. Vir- 
tual keyboards are commercial products offered by several 
35 manufacturers, e.g. Canesta, Senseboard Technologies and 
Samsung. 
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The issue of interacting with virtual graphics has been ad- 
dressed, for example, in the film "Minority report". There is a 
scene where Tom Cruise uses his hands to interact with virtual 
5 information visualized on a Plexiglas "wall". He further uses ges- 
tures for the interaction. 

US 6,614.422 discloses a digital user input to a companion sys- 
tem such as a PDA, a cell telephone or an appliance device, us- 

10 Ing a virtual input device such as an image of a keyboard. A 
sensor captures three-dimensional positional information as to 
location of the user's fingers in relation to where keys would be 
on an actual keyboard. This information is processed with re- 
spect to finger locations and velocities and shape to determine 

15 when virtual keys wou|d have been struck. The processed digital 
information is output to the companion system. The companion 
system can display an image of a keyboard, including an image 
of a keyboard showing user fingers, and/or alphanumeric text as 
such data is input by the user on the virtual input device. 

20 

US 6,618,425 disclose a virtual laser operator. A laser controller 
interconnected with an electric discharge laser communicates 
with a remote computer incorporating a display screen pro- 
grammably emulating a conventional keyboard. The display 
25 screen has a plurality of imaged virtual keys each programmably 
emulating a physical key of a conventional keyboard. A key-, 
stroke is typically applied by manually pressing the position of a 
corresponding virtual key on a touch sensitive screen, or alter- 
natively by actuating a conventional pointing device. 

30 

OBJECTS AND SUMMARY OF THE INVENTION 

The object oif the present invention is to provide a portable con- 
trol panel that the user can carry with him and which almost 
35 weighs nothing. 
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This object Is achieved by means of the initially defined system, 
characterized in that that the system further comprises: a port- 
able identification element, a second tracking unit adapted to 
capture data representing the position of the identification ele- 

5 ment, a storage unit, storing at least one pre-defmed graphical 
Interface representing a control panel of a device, said graphical 
interface comprising an interface for user interactions with the 
device, a graphics unit, generating a graphical representation of 
the control panel based on said stored graphical interface, a 

10 registering unit, registering said graphical representation of the 
control panel in a fixed relation to said portable identification 
element, based on said data representing the position of the 
identification element, to produce a virtual control panel, a dis- 
play unit, showing the user a view comprising the real world and 

15 the virtual control panel projected in a fixed relation to said 
portable Identification element, and an application unit, perform- 
ing actions in response to the users interactions with the virtual 
control panel, and determining which actions to be performed 
based on the position of said user controlled pointing object in 

20 relation to the identification element. 

Preferably, said portable Identification element is adapted to be 
carried by the user during interaction with the virtual control 
panel. The user for example carries the identification element In 
25 his hand or attached to his body. 

The identified device can also be a system. 

The weight of the virtual control panel and the fact that it can be 
30 attached to the body, e.g. an arm or a watch, are clear advan- 
tages of the virtual control panel. 

Preferably, said user controlled pointing object is a handheld 
pointing device or a part of the user's body. 

35 
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According to an embodiment of the invention said graphics unit, 
is adapted to modify said graphical representation of the control 
panel in response to interactions between .the user controlled 
pointing object and the virtual control panel. This embodiment 
5 provides the user with the ability of interaction with the control 
panel, for example scroll and look into menus. 

According to an embodiment of the invention said graphical in- 
terface is adapted for displaying data from the device and that 
10 the system is adapted to generate a graphical representation of 
the data and displaying the data on the virtual control panel, in 
response to interactions between the user 'controlled pointing 
object and the virtual control panel. 

15 According to an embodiment of the invention, the storage unit is 
adapted for storing a plurality of graphical interfaces, each rep- 
resenting a control panel of a particular device, that the system 
is adapted to generate and display a plurality of graphical repre- 
sentations of control panels for different devices based on said 

20 stored graphical interfaces of the devices and that the system 
comprises means for identifying which of the stored control pan- 
els to be displayed. 

According to an embodiment of the invention said means for 
25 identifying which of the stored control panels to be displayed 
comprises a recognition unit for recognizing and identifying de- 
vices in the environment of the user and that the graphics unit is 
adapted to generate a graphical representation of the control 
panel of an identified device based on the stored graphical inter- 
30 face of the identified device. For example said recognition unit 
is adapted for recognizing and identifying unique Identification 
markings on the devices. This embodiment provides the human 
operator with a virtual control panel that adapts its "look" and 
content to the device/system which the operator stands in front 
35 of, or closest to. 
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According to an embodiment of the invention the system is ar- 
ranged so that it automatically changes the virtual control panel 
displayed when another device Is recognized and identified. 

5 According to an embodiment of the invention said display unit 
comprises a head mounted display showing the user said view. 
In another embodiment the view is projected on the eyes of the 
user. 

10 A further object of the invention is to provide a method that en- 
ables a user to walk around and interact with a virtual control 
panel using a user controlled pointing object. 

This object is achieved by the method of the invention as de- 
15 fined in claim 12. 

According to an aspect of the invention, the object is achieved 
by a computer program directly loadable into the internal mem- 
ory of the computer or a processor, comprising software code 
20 portions for performing the steps of the method according to the 
invention, when said program is run on a computer. The com- 
puter program product is provided either on a computer readable 
medium or through a network, such as the Internet. 

25 According to another aspect of the invention, the object is 
achieved by a computer readable medium having a program re- 
corded thereon, when the program is to make a computer per- 
form the steps of the method according to the invention, and 
said program is run on the computer. 

30 

This invention describes a system and a method for a virtual 
control panel that adapts its look and content to specific de- 
vices/systems, for example to the closest system/device. The 
virtual control panel is implemented as augmented reality (AR), 
35 that is computer-generated graphics combined with the real 
worid. The field operator "carries" the virtual control panel with 
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him. The virtual control panel is attached to, for example, a 
unique ID, which is recognized by a tracking system. The field 
operator preferably wears glasses or HMD (head-mounted dis- 
play) on which the computer-generated graphics are projected 
5 (either as optical see-through or video see-through). The virtual 
control panel only exists as data projections. The virtual control 
panel represents the user interface for interacting with process 
devices and/or systems. Examples are status monitoring, set 
point changing, reports and other textual documents, historical 
10 data in form of trend curves, 3D simulations, interaction buttons 
and menus. The virtual control panel adapts its look/Interface to 
the related device/system (may also be identified with unique 
IDs recognized by the tracking system). 

The virtual control panel changes Its look and functionality re- 
garding which device/system it is close to. A recognition unit de- 
tects the identity of the closest device/system. The AR system 
visualizes the related virtual control panel, which may provide a 
range of different Information and information presentations. The 
field operator interacts with the virtual control panel using an 
input device or his fingers/hands. Also, other interaction means 
may be used such as voice and physical interaction buttons / 
haptic means. The field operator navigates in the different views 
of the virtual control panel. When the operator moves towards 
another device/system, the virtual control panel changes its look 
to represent the new device's/system's functionality and charac- 
teristics. The representations of the different virtual control pan- 
els are defined in advance and stored at a computer or a server. 
In addition to historical data, real-time data and information is 
visualized on the virtual control panel. 

In one use-case, an operator (a system or process expert) work- 
ing within the plant needs to check and update an existing robot 
program to make sure that the program modules are 
35 downloaded and related to the right robots (four robots included 
in the program). He then has to log in at the each of the four 



15 



20 



25 



30 
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teach pendants, one at a time, scroll down In the menus and find 
the right program and the right place m the program. By utilizing 
the proposed invention, the operator would only need to walk 
next to the appropriate robot and then check the program for 
5 each of the four robots. Costs related to maintaining the teach 
pendants will be reduced. Also, the problem with wires between 
the robot controller and the teach pendant is avoided with the 
virtual control panel. 

10 In process plant environments the operator walks around and 
performs monitoring, inspection and maintenance tasks related 
to a process or device. The operator typically has two-ways ra- 
dio communication with the control room operators to transfer 
information to and from the process. The virtual control panel 

15 provides the field operator with an interface for online Interact- 
ing with the various devices and systems in the field as well as a 
tool for viewing documentation, reports, maintenance and ser- 
vice procedures, etc. The operator always carries the virtual 
control panel with him. He can put it aside, or attach it, e.g. to 

20 his arm, while doing other tasks. The field operator will be able 
to perform his tasks more efficiently. Also, the use of the virtual 
control panel prevents errors due to communication problems 
between the field operator and the control room (misunderstand- 
ings, noise, etc.). The control room operators may also have the 

25 opportunity to see updated data and information from the plant. 

The virtual control panel can be used for other applications such 
as registration and counting in stores. Combined with bar codes, 
the virtual control panel may visualize information related to the 
30 registered part, etc. The user can enter data and information as 
well as reading status and documentation. Office tool 

Another example application is an office. The virtual control pa- 
nel may provide the user with names of people by passing the 
35 doors, or the people themselves if the employees are equipped 
with IDs. For example, the security staff may have virtual control 
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panels for checking IDs, changing passwords, recording status 
Information from their check rounds, etc. 

Advantages gained by this invention is that the field operator 
5 always has the control panel with him, easy standardization of 
the various user interfaces, the operator only interacts with a 
single interface which changes its representation (look and con- 
tent) regarding the closest functionality and characteristics de- 
vice/system, the virtual control panel almost weights nothing. 
10 The field operator can put the virtual control panel aside while 
he is doing other tasks, e.g. put it down on a table, the floor, 
etc. The virtual control panel Itself has no need for rugge- 
dised/inthnsic safe certification. 

15 BRIEF DESCRIPTION OF THE DRAWINGS 

The invention will now be explained more closely by the descrip- 
tion of different embodiments of the Invention and with reference 
to the appended figures. 

20 

Fig. 1 shows a system for providing a virtual control panel ac- 
cording to an embodiment of the Invention. 

Fig. 2 shows a block scheme over a system according to the 
25 Invention. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
OF THE INVENTION 

30 This invention proposes a system and a method for providing a 
virtual control panel which adapts its interface and content to a 
device and/or a system. The system comprises the following: 
A handheld interacting and. pointing device with a tracking sys- 
tem for determining its position and orientation in relation to a 

35 world coordinate system. 
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A handheld or otherwise wearable unique ID including a tracking 
system for determining its position and orientation in relation to 
a world coordinate system. 

A wearable display device (glasses, head-mounted display or 
5 head-up display) for visualizing augmented reality overlaid the 
view of the real world. 

Unique IDs for placing at objects (devices and systems) in the 
environment Including a recognition system for recognizing the 
unique IDs. 

10 In case video see-through Is used for augmented reality visuali- 
zation, the display device further comprises: 
A camera for capturing a stream of images of the environment 
mounted on or integrated with the display device. The camera 
will be mounted in a fixed position at the display device, and the 

15 display device will be located along the camera view axis and at 
the camera's image plane. 

A system for generating an augmented reality representation of 
computer-generated graphical information overlaid the real 
world. 

20 The system can be configured In different ways. 

The two most likely ways of visualizing augmented reality are 
Video see-through and Optical see-through. Video see-through 
needs a camera to capture a live video stream of the environ- 

25 ment corresponding to the user's view of the world. The system 
combines computer-generated graphics with the live video 
stream and projects the combined augmented reality video onto 
the display device. The user will see the video with overlaid vir- 
tual information as if he/she was looking at the real world. For 

30 optical see-through, the computer-generated graphics are regis- 
tered directly onto (combined with) the display device and fol- 
lows the user's view of the real world. The virtual graphics are 
overlaid the real world without including a video of It. The former 
solution is less demanding regarding the frame rate. 

35 
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Basically, two tracking systems are needed for the virtual control 
panel system in addition to a recognition system for recognizing 
IDs attached to the devices/systems in the environment. The 
first tracking system traces and determines the interacting and 
5 pointing device whereas the second tracking system traces and 
determines the ID held by the user. These tracking systems may 
be a single tracking system used for tracing both, or two differ- 
ent tracking systems. A vision-based tracking system including a 
camera mounted on, or Integrated with, the display device is one 
10 solution which may be combined with AR visualization based on 
video see-through. Examples of tracking systems are ultrasonic 
tracking systems, magnetic tracking systems, tracking systems 
based upon inertial navigation (using accelerometers and gy- 
ros), mechanical arms, vision-based and optical systems and 
15 hybrid systems combining technologies previously mentioned. 
For some of the tracking systems, the interacting and pointing 
device and the handheld/wearable ID (held by the user) may 
have integrated sensors for determining positions and orienta- 
tions. Such sensors may, for example, be gyros, Inertial sensors 
20 and accelerometers. 

The origo of the world coordinate system is defined to corre- 
spond with the display device. As the camera is mounted on, or 
integrated with, the display device, the relation between the 
25 camera and the display device Is known. Hence, the system de- 
termines the poses of the interacting and pointing device and 
the handheld ID In relation to the camera/display device system. 

The system also needs to recognize the predefined IDs in the 
30 environment. These IDs are attached to devices and systems 
which the operator wants to interact with. Different technologies 
can be used for the recognition such as RF tagging, bar codes, 
ultrasound and GPS. Alternatively, a tracking system as de- 
scribed above can be used to recognize/trace the positions of 
35 the device/system IDs. 
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The user wears a display device (either wearable glasses or a 
head-mounted display/ head-up display), a wearable computer, 
a handheld or otherwise wearable unique ID, and an interacting 
and pointing device. 

5 

First, a fixed world coordinate system needs to be defined. The 
^ origo of this coordinate system is aligned with the display device 
so that all pose information provided by the tracl<ing systems is 
related to this world coordinate system. 

10 

In advance, personnel place predefined unique IDs at devices 
and systems in the environment, which the operator needs to 
interact with. In relation to each device and system, virtual inter- 
faces need to be defined in the system. Also, potential commu- 
15 nication between the virtual Interfaces and the devices/systems 

ft 

must be defined and set up in advance. 

When the operator starts using the system, he walks around in 
the environment until he comes to a device/system which he 

20 wants to interact with. He moves the handheld/wearable ID next 
to the device/system until the system has recognized the spe- 
cific device/system. The related virtual interface appears as a 
virtual control panel attached to the handheld/wearable ID. The 
operator uses the interaction and pointing device to interact with 

25 the virtual control panels, which may include interaction buttons, 
menus and other types of interaction possibilities. The virtual 
control panel updates its look and content corresponding to the 
operator's choices. Alternatively, the operator may use his 
hand(s) and finger(s) for interacting with the virtual control 

30 panel. The system still needs to determine the pose of the 
interaction body part. A similar tracking system as for the 
interacting and pointing device can be used for hand/finger 
tracking. The system further needs a recognition system in order 
to recognize the interacting body part (hand{s) and finger(s)) 

35 and determine its position. 
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The virtual control panel changes its look and content as the 
operator either moves along to a new device/system, which has 
attached an ID, or interacts with the virtual control panel. Exam- 
ples of the virtual interfaces are device/system overviews, status 
5 information, set points, trend curves, documentation, video clips 
,e.g. for maintenance and procedures, and data entry. 

The virtual control panel can be combined with other interaction 
modalities such as voice and tactile feedback. Typically, the vir- 
10 tual control panel may give an audio feedback when the opera- 
tor pushes an interaction button. 

The operator can move the handheld/wearable ID freely (both 
regarding position and orientation) and the virtual control panel 

15 will follow the ID as it is physically attached to the ID. The re- 
lated tracking system captures the movements of the hand- 
held/wearable ID and transmits the pose to the wearable com- 
puter by means of a communication link, e.g. wireless communi- 
cation. Hence, the operator can put the virtual control panel 

20 away (e.g. in a pocket) when he/she is not using it, or put the 
virtual control panel on a table, etc. when doing other tasks. 
The wearable computer communicates by wireless communica- 
tion with the devices/systems, e.g. to retrieve and send data. 
Readings from the devices/systems may be presented on the 

25 virtual control panel whereas the operator may modify, e.g. the 
set point of a device/system. 

Video see-through is used to generate and present an AR world 
at the handheld/wearable ID representing the virtual control pa- 

30 nel. The camera integrated with the display device is used to 
capture a live video stream of the real world. The camera Is lo- 
cated in relation with the displiay in such a way that it provides 
the same view as the user would get by looking directly at the 
real world. The live video stream combined with computer- 

35 generated graphics is presented in real-time at the display de- 
vice. Additional functionality includes camera zooming with oui- 
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put of the actual camera focal length. This will enable the sys- 
tem to display the computer-generated graphics correctly while 
zooming. The camera may also be used for vision-based track- 
ing if vision-based tracking is used as the tracking system. 

5 

Figure 1 shows an embodiment of a system according to the in- 
vention. The operator (1) walks around in the environment. The 
tracking system (6) recognizes known IDs (4, 9) in the environ- 
ment. The operator holds an ID (4) in his hand or wears it on his 

10 body, e.g. around his hand wrist. In case the tracking system is 
vision-based, the operator wears a camera (7) attached to his 
head. The camera (7) records live video of the real world. The 
camera (7) may also be used for the augmented reality visuali- 
zation of the virtual control panel in case the "video see- 

15 through" method is used. 

The operator (1) carries an ID (4), which will provide him/her 
with an augmented reality view of the control panel. The virtual 
control panel is visualized either as "optical see-through" or "vi- 

20 deo see-through" at the wearable display (3) (wearable glasses 
or head-mounted display/head-up display). The user has an in- 
teracting and pointing device (5) for interacting with the virtual 
control panel. The interacting and pointing device (5) may have 
an integrated tracking system making it possible to point at 

25 poses in the environment and to interact with the system. 
Alternatively, the camera (7) may be used as vision-based 
tracking of the interacting and pointing device. 

Devices/systems (8) in the environment have some type of IDs 
30 (9) attached so that the tracking system (6) recognizes the spe- 
cific device/system. The tracking system (6) determines the po- 
sition and orientation of the devices/systems based on the IDs 
(9). The IDs (9) haye corresponding virtual interfaces which are 
presented as the virtual control panel. 

35 
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A wearable computer (2) or a stationary computer contains the 
necessary software in order to generate the augmented reality 
control panel based upon the output from the tracking system 
(6) and the video stream from the camera (7) (in case video see- 

5 through is used). The wearable computer (2) also contains the 
necessary software needed to perform the desired task or proc- 
ess, e.g. (two-ways) communication with devices/systems, real- 
time data managing and visualisation. Further, the wearable 
computer (2) will generate the graphics, which will provide the 

10 AR interface view. Finally, the wearable computer (2) will con- 
tain a storage media in order to. save, and restore previously sa- 
ved, information. 

Communication of the information from the interaction and point- 
15 ing device (5) to the wearable computer (2) is done through a 
wired or wireless link. Depending on the type of tracking system, 
the pose of the interacting and pointing device (5) can come 
from the pointing device itself, or from an external tracking sys- 
tem. Communication is performed through a wired or wireless 
20 communication media. 

Figure 2 shows a block scheme over the system. The application 
unit (21) contains the necessary software in order to perform the 
desired process or task. e.g. integrate real-time data with rec- 

25 ognized device/system, and perform actions based on input from 
the interacting and pointing device. The tracking system will 
make it possible to trace the pose of the interacting and pointing 
device (5) In 3D. and can also be used for specifying the world 
coordinate system. The interacting and pointing device can also 

30 be used for additional tasks such as to drag a virtual control 
panel, or other virtual information, from the handheld ID to a 
specific device/system and to locate, select, show, etc. new po- 
sitions in the environment. Further, the application unit holds 
information regarding the coordinate systems. Hence, the appli- 

35 cation unit (21) Is dependant on the specific sys- 
tem/environment. 
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The operator uses an Interaction and pointing device (5) whicli 
are traclced by a tracl<ing system (20) wiiicli may be integrated 
with the interacting and pointing device. The user uses the in- 
teracting and pointing device (5) for interacting with the virtual 
control panel and for pointing in the environment. 

A graphics unit (23) generates a 3D graphical representation of 
the visual information that is to be displayed in relation to the 
user's ID (4). That is, this unit holds a specification of the 3D 
graphical primitives to be visualized. The graphics unit (25) re- 
ceives information from the application unit (21) regarding the 
virtual graphical information to be displayed based on user in- 
put. All graphical primitives positions are specified in relation to 
the world coordinate frames. For the robot programming applica- 
tion relevant visual information may be operator-specified way- 
points, the actual robot path, task specific information etc. The 
graphical representations in the graphics module are transmitted 
to the registration unit (25). 

The registering unit (25) combines the virtual graphics from the 
graphics unit (23) with the real world view. In case the video 
see-through method is used, a live video of the real world image 
from the camera system unit (26) is captured and combined with 
the graphical representation from the graphics unit (23). The 
registering unit (25) uses the pose data from the ID tracking sys- 
tem unit (24) in order to overlay the real world Images with the 
computer-generated graphics. With the registration correctly 
done, the computer-generated graphics will be virtually "at- 
tached" to the real world scene. 

In case video see-through, a camera unit (7) attached to the 
display device (3, 27) produces real world images. The pose of 
the camera and display is determined according to the world co- 
ordinate frame, and is used by the registering unit (25) in order 
to overlay the real world scene with computer-generated graph- 
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ics from the graphics module (23). The combined images are 
displayed on the display device (27). Alternatively, the register- 
ing unit (25) overlays the real world with the computer- 
generated graphics (optical see-through). 

A storage unit (22) enables saving and loading of application 
related information, e.g. application related information, graphi- 
cal interfaces for the various devices/systems, and system con- 
figuration parameters (e.g. local and world coordinate frame). 

The present invention is not limited to the embodiments dis- 
closed but may be varied and modified within the scope of the 
following claims. For example 



